Skip to content

feat(docker): support explicit WSL all-GPU injection - #4854

Draft
rsliter wants to merge 11 commits into
NVIDIA:codex/1606-cdi-policy-dockerfrom
rsliter:feat/4849-wsl-gpu-compat/rsliter
Draft

rsliter wants to merge 11 commits into
NVIDIA:codex/1606-cdi-policy-dockerfrom
rsliter:feat/4849-wsl-gpu-compat/rsliter

Conversation

@rsliter

@rsliter rsliter commented Oct 22, 2026

Copy link
Copy Markdown

Summary

Support explicit all-GPU NVIDIA injection through ordinary Docker Desktop WSL sandbox creation. Preserve OpenShell ownership of the workload, protected GPU selection and restart admission while keeping existing CDI behavior unchanged.

Depends on #4204; the tested parent revision is 33350a881f55011286e7c3a68d8007e43e50e0bf.

Related Issue

Closes #4849

Changes

  • Add operator-gated docker.nvidia_wsl_gpus=all, limited to Docker Desktop WSL and rejecting conflicting CDI or counted GPU requests.
  • Create the NVIDIA DeviceRequest through the Docker driver and bind selection to protected boundary/runtime metadata.
  • Validate persisted selection during admission and restart.
  • Document configuration and update CLI guidance; cover eligibility, selection conflicts and admission mutation cases with regression tests.

Testing

  • Docker driver suite: 165 passed on macOS and ARM64 Linux; two existing tests ignored.
  • Strict Clippy and Rust formatting passed.
  • Real ARM64 N1X Docker Desktop WSL: non-root CUDA initialization, enumeration, allocation, PTX kernel execution and result retrieval passed.
  • Filesystem/network denial, seccomp, no-new-privileges and zero effective capabilities observed.
  • Sandbox stop/start and gateway restart preserved exact workload/GPU identity, state and CUDA behavior.
  • Owned workload/supervisor cleanup completed and QA gateway stopped.
  • Documentation navigation and Markdown lint passed. Fern returned zero errors; all three warnings reproduce on the unchanged parent (MDX parsing, unauthenticated redirects check and theme contrast).
  • Repository CI and automated review completed.

Hardware evidence: #4837 (comment)

The test used the local uncommitted patch on the parent revision, identified by compressed patch SHA256 9791d6251efe4f958d460d62cb9cbc5f09646edcd0a1d0c5683455a580683c71. Local commit c0730ecb8c9c8a4562af534ca717af72a06ddd61 matches the tested implementation patch; its only additional change is runtime-overview documentation. The SSH signature was verified locally. Hardware: RTX Spark N1X, Ubuntu24.04 WSL2, Docker Desktop4.96.0 / Engine29.9.1, NVIDIA620.66. Development gateway transport was used. Jetson, x86 hardware, individual GPU selection, model inference and production transport are not qualified by this evidence.

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture/configuration guidance updated

Draft review context

Opened as a draft following Jim Meyer's invitation to share the proposed approach for review, relayed by the author. Issue #4849 still awaits state:accepted; the author currently has Read access to the upstream repository. The issue-acceptance check may remain red until maintainers record acceptance. This draft does not claim acceptance or merge readiness.

The PR targets #4204's branch to isolate this change. Retarget and reconcile validation after the dependency lands; the closing reference takes effect when this work reaches the default branch.

elezar and others added 11 commits October 15, 2026 16:52
Signed-off-by: Evan Lezar <elezar@nvidia.com>
Have the openshell client role publish the installed gateway's name,
active TOML path, service scope, service user, and network, and add an
openshell_test_gateway role that loads that context, resolves the CLI,
restarts the gateway service, and collects its journal. A feature
suite can then change the gateway's active configuration for its own
run and restore it afterwards.

Select the candidate runtime images for packaged installs through a
qualification gateway.toml referenced by OPENSHELL_GATEWAY_CONFIG
instead of OPENSHELL_* variables in the service environment file, so
the gateway has a TOML file a suite can modify. Update the packaging
asset check to match.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Define each CDI scenario once in vars/scenarios.yaml: the host
fixtures it needs (directories, files, a character device, a socket),
the CDI edits that reference them, and the behavior the workload is
expected to observe. The cdi_fixtures role creates the fixtures,
renders one tmachine.example/cdi-scenario spec with a device per
scenario, and publishes the expectations as scenarios.json.

Install the spec into /var/run/cdi from systemd-tmpfiles on every
boot so it survives tmachine's setup, installation, and test disk
layers, and run the role as part of the ubuntu-docker-rootful setup.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Add a cdi feature suite that creates a sandbox for each CDI scenario
by selecting its device explicitly with --gpu 1 and the Docker
cdi_devices driver option, then checks what the workload observes or
that creation is rejected. The sandbox only reports facts; the test
makes every assertion, so a failure names the check that disagreed.
Each scenario is its own test.

The expectations record the behavior the sandbox has today, loaded
from the scenarios.json published by the fixtures role. A
OPENSHELL_CDI_CAPTURE mode prints observations instead of asserting,
to record that behavior for a new scenario.

The test playbook enables caller driver configuration in the installed
gateway's active TOML for the run and restores it afterwards. Build the
suite's nextest archive in tests/artifacts.nix, expose it as
build-artifacts-cdi-tests, and run the suite on the Docker environment
with the deb installer in the branch and release matrices.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Define the shared CDI context schema: the selected CDI device IDs and
the workload-local paths at which a compute driver projects CDI spec
directories. Carry it as an optional field in the sandbox boundary
configuration, omitted from serialized output when absent so non-GPU
bootstraps stay readable by older sandbox runtimes. No driver sets it
yet.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Resolve the selected CDI devices against the spec directories projected
into the workload and derive the filesystem requirements their edits
imply:

- device nodes, which need read-write access;
- read-only mounts, expanded to the parent directory for shared
  libraries;
- Unix sockets, kept as exact targets without filesystem grants;
- the NVIDIA MPS pipe directory, accepted as a writable IPC directory
  only when mounted nosuid, nodev and noexec;
- additional GIDs, excluding root.

Effective mount access comes from the workload's mount table rather
than spec options. Ordinary writable mounts, broad roots and
conflicting uses of one path are rejected.

Captured Orin, Spark and multi-GPU specs drive the production resolver
against reviewed baselines. The resolver is wired into launch in the
next commit.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
When the boundary configuration carries a CDI context, resolve it at
agent start and use the result instead of the hard-coded GPU baseline:

- add derived read-only paths, and read-write access for device nodes
  and IPC directories, unless the authored policy already covers them;
- reject an authored read-only entry for a device node or IPC directory
  instead of overriding it;
- validate every CDI target through the descriptor that anchors its
  Landlock rule at each launch and exec, rejecting broad-root aliases,
  type changes and changed mount access;
- require Landlock enforcement for CDI targets even when the policy is
  best effort.

Missing CDI additionalGids are reported as a warning, because some
runtimes do not apply them and devices that do not depend on them stay
usable. A required root group remains fatal. Enrichment, failures and
the effective Landlock mode are reported as OCSF events.

Sandboxes without a CDI context keep the existing GPU baseline.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
For GPU sandboxes, record the selected CDI device IDs and the spec
directories the Docker daemon currently reports in the protected
boundary configuration, and project those directories read-only under
/.openshell/cdi-specs so the sandbox runtime can resolve the same
specs. User mounts, image volumes and the workspace cannot overlap the
projection path.

Before creating the sandbox, probe each directory with a stopped
container from the prepared workload image and skip directories that
do not exist on the Docker host. Probes are bounded by a timeout,
labelled for cleanup with their sandbox, and remove their anonymous
volumes. A probe that cannot be removed no longer fails creation.

With resource admission enabled, containers with a CDI context must
keep a CDI device request matching the recorded selection and one
read-only, non-recursive, private bind per recorded directory.
Admission reads the bootstrap only for CDI containers, and boundary
files are replaced atomically so it never observes a partial write.

The Docker e2e gateway enables resource admission in GPU mode.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Describe how Docker GPU sandboxes derive filesystem access from the
selected CDI devices, which CDI mounts are rejected, the NVIDIA MPS pipe
directory exception, missing additionalGids handling on Docker releases
before 29.5, and SELinux requirements for the projected spec
directories. Add cluster debugging entries for CDI projection,
admission, probe and restart failures, and document the CDI context in
the core, sandbox and Docker driver READMEs.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Record the behavior of the named scenarios once filesystem access is
derived from the selected CDI specs: read-only mounts and library parents
are readable, device nodes are writable, and writable mounts, an unsafe
library parent, and an NVIDIA MPS directory without noexec are rejected.
Add scenarios for the MPS pipe directory with and without noexec.

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Keep NVIDIA injection under Docker driver ownership and validate the persisted selection during admission and restart.

Closes NVIDIA#4849

Signed-off-by: Rebecca Sliter <571084+rsliter@users.noreply.github.com>
@copy-pr-bot

copy-pr-bot Bot commented Oct 22, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions

Copy link
Copy Markdown

Thank you for your submission! We ask that you sign our Developer Certificate of Origin before we can accept your contribution. You can sign the DCO by adding a comment below using this text:


I have read the DCO document and I hereby sign the DCO.


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the DCO Assistant Lite bot.

@rsliter

rsliter commented Oct 22, 2026

Copy link
Copy Markdown
Author

Thank you for your submission! We ask that you sign our Developer Certificate of Origin before we can accept your contribution. You can sign the DCO by adding a comment below using this text:

I have read the DCO document and I hereby sign the DCO.

You can retrigger this bot by commenting recheck in this Pull Request. Posted by the DCO Assistant Lite bot.

I have read the DCO document and I hereby sign the DCO.

@rsliter

rsliter commented Oct 22, 2026

Copy link
Copy Markdown
Author

WSL hardware validation

Tested implementation: c0730ecb8c9c8a4562af534ca717af72a06ddd61.

A fresh qualification on Docker Desktop WSL2 passed for the explicit docker.nvidia_wsl_gpus=all path. Before running, we fetched the published commit and verified that the remote source matched its implementation, including the new WSL GPU module. We reused the previously built ARM64 binaries after verifying all four recorded SHA-256 hashes; this was not a new build from the published commit. The additional runtime overview documentation does not change those binaries.

Environment: NVIDIA RTX Spark N1X, WSL Ubuntu 24.04 ARM64, kernel 6.18.33.2-microsoft-standard-WSL2, Docker Desktop 4.96.0 / Engine 29.9.1, NVIDIA driver 620.66. Workload image: ghcr.io/nvidia/nemoclaw/sandbox-base:v0.0.137, digest sha256:d8bed276c8ae089f4e1fc421288e1e793b36740a9d7f594586dddb52b8de2505.

Stage Result
Normal sandbox creation with explicit GPU selection Passed
CUDA initialization, device enumeration, allocation, PTX kernel execution, and result readback Passed: one device, kernel result 42
Sandbox stop/start CUDA, isolation, and persisted sandbox state passed
Gateway restart Same sandbox recovered to Ready; CUDA, isolation, and persisted state passed
Cleanup Sandbox deleted, sandbox list empty, no running Docker containers, QA gateway exited and port released

Each CUDA probe returned:

{"uid":998,"devices":1,"kernel_result":42,"cuda":"pass"}

Enforcement probes confirmed denied writes outside permitted paths, denied direct network access, zero effective capabilities, NoNewPrivs: 1, and seccomp mode 2. Strict Landlock remained enabled. The Docker Desktop environment does not advertise AppArmor, so this does not establish AppArmor coverage.

The script initially attempted exec while gateway recovery still reported Provisioning and exited. After confirming that the same sandbox reached Ready, we continued the remaining probes successfully without recreating it or repeating the gateway restart. The early failed probe and successful continuation logs are retained separately.

Raw qualification logs are retained on the test host under ~/openshell-4854-evidence. This establishes the tested single-GPU ARM64 WSL path with all selection. It does not establish x86 WSL, Jetson, multi-GPU selection, inference, or production transport qualification; the dedicated QA gateway used local development transport without provider credentials.

GitHub validation remains separate: Branch Checks and Helm Lint await maintainer /ok to test c0730ecb8c9c8a4562af534ca717af72a06ddd61 admission. The current E2E/GPU E2E success statuses indicate their opt-in labels were absent, not that those suites executed.

@rsliter

rsliter commented Oct 23, 2026

Copy link
Copy Markdown
Author

recheck

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

This pull request has had no activity for 14 days and is now marked stale.

@github-actions github-actions Bot added the state:stale Inactive item at risk of automatic closure. label Nov 7, 2026
@elezar
elezar force-pushed the codex/1606-cdi-policy-docker branch from 33350a8 to baf423b Compare November 16, 2026 11:42

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state:stale Inactive item at risk of automatic closure.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants